Australian Journal of Teacher Education 


Volume 42 Issue 10 


Article 4 


2017 


Commercially available Digital Game Technology 
in the Classroom: Improving Automaticity in 
Mental-maths in Primary-aged Students. 


John O'Rourke 

Edith Cowan University, j.o_rourke(2>ecu.edu.au 


Susan Main 

Edith Cowan University, s.mainffflecu.edu.au 

Susan M. Hill 

Edith Cowan University, susan.hill(®ecu.edu.au 


Recommended Citation 

O'Rourke, J., Main, S., & Hill, S. M. (2017). Commercially available Digital Game Technology in the Classroom: Improving 
Automaticity in Mental-maths in Primary-aged Students.. Australian Journal of Teacher Education, 42(10). 
http://dx.doi.Org/10.14221/ajte.2017v42nl0.4 


This Journal Article is posted at Research Online, 
http: / / ro.ecu.edu.au/ajte/vol42/iss 10/4 





Australian Journal of Teacher Education 


Commercially Available Digital Game Technology in the Classroom: 
improving Automaticity in Mental-Maths in Primary-Aged Students. 


John O’Rourke 
Susan Main 
Susan Hill 

Edith Cowan University 


Abstract: In this paper we report on a study of the implementation of 
handheld game consoles (HGCs) in 10 Year four/five classrooms to 
develop student automaticity of mathematical ccdculations. The 
automaticity of mathematical calculations was compared for those 
students using the HGC and those being taught using traditional 
teaching methods. Over a school term, students (n=236) who used the 
HGCs and Dr Kawashima’s Brain Training showed significant 
improvement in both the speed and accuracy of their mathematical 
calculations. Data collected in interviews during the intervention 
period from students, staff and parents were analysed to provide 
further information on the implementation and efficacy of this 
approach. This exploration identified that the HGCs contributed to 
positive learning, motivational, and efficiency outcomes. These 
findings highlight opportunities for using commercially available 
digital games to achieve classroom objectives. 


Keywords 
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Let’s accept the inevitability of things to come - and play with them! 
(Thiagarajan, 2001, p. xiii) 


Introduction 

Is it not inevitable that children who have grown up using digital games will expect 
education and play to be intertwined? To appreciate this sentiment one only has to observe 
toddlers in prams using phones and tablets and project forward. Surely these toddlers will 
enter school having a much different view of digital technology than their parents or others 
from previous generations. Such speculation has probably triggered the research interest in 
the role of digital tools (particularly in the guise of games) in the educational lives of students 
(Gee, 2003; Oblinger & Oblinger, 2005; Prensky, 2001, 2006, 2010; Rosas, Nussbaum, 
Cumsille, Marianov, Correa, et al., 2003; Rosen, 2010) and learners in general (All, Nunez 
Castellar & Van Looy, 2014, 2015, 2016; Nunez Castellar, Van Looy, Szmalec & De Marez, 
2014, 2015). Indeed, Prensky (2001, p. 2) hypothesised that digital game-based learning 
(DGBL) and its more ‘sophisticated successors’ will be taken for granted by learners in the 
future. 
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Whereas Prensky and like-minded scholars predict the ubiquity of DGBL in schools, 
many educators are resistant to this viewpoint (Logie & Della Sala, 2010; Purcell, 2005). So 
the research interest in ‘game playing’ within DGBL is both the basis for optimism and for 
cynicism - the latter in terms of cost, distraction from real learning objectives, social isolation 
and reduced attention spans (Marquis, 2013). To explore the potential of DGBL for mental- 
maths skill development we conducted a pilot study involving 59 primary students using 
HGCs (Handheld Game Consoles) utilising Dr Kawashima’s Brain Training program 
(designed to develop mental alertness through a series of activities). The finding indicated 
that students using HGCs made significantly more improvement in mental-maths 
automaticity than those who developed skills via typical classroom approaches (Main & 
O’Rourke, 2011). In this article, we report on an extension of the earlier study. The number 
of students and primary schools was increased so that stronger generalisations about the 
capacity of digital tools to enhance mathematical automaticity might be made from the 
findings. Lurther, we interviewed students involved in this study, along with their parents and 
classroom teachers, at regular intervals over a school term to elicit their thoughts on whether 
the HGCs were easy to implement, engaging, and enjoyable for students, and whether the 
students’ parents were satisfied that this was a positive approach for developing mental-maths 
skills. This paper reports on the student achievement data for the project and then draws 
together the authors previously published research on the qualitative elements of this 
research, subjecting these data to additional analysis in order to provide a comprehensive 
picture of the impact of the intervention. 


Literature review 

Digital Games Based Learning (DGBL) in the Classroom 

Several researchers have identified the efficacy of DGBL in classrooms (for example: 
Clark, Tanner-Smith, & Killingsworth, 2016; Condie & Simpson, 2004; Erhel & Jamet, 
2013; Groff, Howells, & Cranmer, 2010; Main & O’Rourke, 2011; Miller & Robertson, 
2009, 2011). However, researchers also found scepticism among academics and practitioners 
that using digital games (particularly those normally associated with leisure activities) is 
worth the time and financial investment (Bennet, Maton, & Kervin, 2008; Logie & Della 
Sala, 2010; Sardone & Devlin-Scherer, 2010), especially when more traditional approaches 
could be just as successful (Prensky, 2001). Concerns about the use of digital technology in 
education often focus on the resulting increased screen time and the associated health risks 
that come with this (Houghton, Hunter, Rosenberg, Wood, Zadow, Martin, & Shilton, 2015). 
As with much change, some classroom practitioners understandably lament for lessons past, 
or as Tapscott (2009, p. 128) points out in his exploration of learning in schools for students 
of the ‘net-generation’; “old paradigms die hard”. A report on digital games in the classroom 
found that teachers who played digital games for pleasure were much more likely to use 
digital games in their classrooms than those who did not (Takeuchi & Vaala, 2014); 
therefore, it is not surprising that some classroom teachers have expressed a need for 
guidance on how to use DGBL effectively (Greenhow, Robelia, & Hughes, 2009; Jukes, 
McCain, & Crockett, 2010; O’Rourke, Main, & Ellis 2013). 


Digital Games and Mathematics 

The school subject that appears to have benefitted most from the intervention of 
digital technology, and specifically DGBL, is mathematics (Miller & Robertson, 2009; 2011). 
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Coley, Cradler and Engel (2000) in an early exploration of technology in American schools 
reported positive outcomes in drill and practice computational programs. McFarlane, 
Sparrowhawk, and Heald (2002) in a large study involving twelve primary and secondary 
schools in the UK, found digital games were effective when developing algebra skills. Ke 
(2008) investigated the use of drill and practice computer games for a small group of Year 
4/5 students (n=15) and found limited testing gains at the completion of the research phase, 
but enhanced motivational attitudes toward maths. Miller and Robertson (2009; 2011) 
investigated the use of HGCs to develop automaticity in mental-maths skills for Scottish Year 
4 and 5 students and found significantly higher scores in both accuracy and speed in mental- 
maths for students using the HGCs than those in comparison classes using non-digital 
methods to develop mental-maths skills. Likewise when we replicated this study on a smaller 
scale, (n=59 students), we found significant differences in mental-maths accuracy and 
enhanced self-concept toward mathematics for students using the HGCs compared to those in 
comparison classrooms (Main & O’Rourke, 2011). 

Chang, Wu, Weng, and Sung (2012) investigated the use of game-based problem 
solving for mathematics with four fifth grade classes in Taipei and found that students had 
more flow experiences, and higher problem-solving and problem-posing abilities than those 
in the comparison group. Another study, involving 193 American secondary students and 10 
teachers, exploring the effects of a 3D computer game on mathematics achievement and 
motivation, found that there were significant improvements in mathematics achievement for 
students using the program compared to a comparison group using a more traditional 
approach (Kebritchi, Hirumi, & Bai, 2010). A relevant finding from Kebritchi et al. (2010), in 
the context of this current study (where the HGCs were embedded in the classroom rather 
than used in a separate setting), was that students who played the games only in the school’s 
computer laboratory reported being less motivated than those who played the game in their 
classrooms and laboratory. A recent review of 60 studies involving mobile technologies for 
mathematics instruction, including HGCs, found that students’ attitude to the use of this 
technology was mostly positive and student engagement with the learning activities increased 
(Fabian, Topping, & Barron, 2016). Further, their meta-analysis of student achievement data 
indicated an effect size of 0.48. 

The performance of Australian students in mathematics continues to be an area for 
concern with the release of the Trends in International Mathematics and Science Study 
(TIMSS) and Programme for International Student Assessment (PISA) data. Data from 
TIMSS indicates that less than 10% of Australian students attain the advanced international 
benchmark (Thomson, Wernert, O’Grady, & Rodrigues, 2017), while PISA results reveal that 
Australia’s performance in mathematics declined significantly between 2012 and 2015 
(Thomson, De Bortoli, & Underwood, 2016). Professor Gordon Stanley, Chairman of the 
Review Panel of the National Numeracy Review Report (Commonwealth of Australia, 2008) 
suggests that part of the reason for continued poor performance of Australian students in 
mathematics; “is [that it is] not generally perceived as a popular subject among young 
people” (p.l), nor is it, “recognised as an easy subject to leam or to teach” (p. 1). 

It has been asserted that one of the biggest challenges to student success is 
disengagement (Appleton & Lawrenz, 2011; Parsons, Nuland, & Parsons, 2014) and, 
arguably, DGBL offers a potential remedy for the apathy experienced in some primary maths 
classrooms. Whereas ‘fun’ has rung classroom alarm bells for educators in generations past 
(Harp & Mayer, 1998), it is now seen as appropriate in well-structured learning environments 
(Elton-Chalcraft & Mills, 2013; Mathers, 2008). There is mounting evidence that the strategic 
and thoughtful implementation of digital games has the capacity to facilitate a fun and 
productive environment for learning mathematics (Kebritchi et al., 2010; Somyiirek, 2015). 
First and foremost, many digital games such as Dr Kawashima ’s Brain Training demand and 


Vol 42, 10, October 2017 


52 



Australian Journal of Teacher Education 


reward improvement, and as such the stimulation, motivation, and learning associated with 
improved game performance are all closely aligned (Hamari, Shemoff, Rowe, Coller, Asbell- 
Clarke, & Edwards, 2016; Jukes et al., 2010). 

Given that classroom teachers are being held increasingly accountable for numeracy 
standards via national standardised testing (for example, NAPLAN, 2016), they must be 
confident that the tools they employ are easy to instigate and clearly focussed on student 
success. In the case of mental-maths automaticity, the Australian Curriculum identifies 
‘understanding and fluency’ as a key proficiency strand in developing primary students’ 
number sense; consequently, strategies that enhance this development are vital. McIntosh, 
Reys and Reys (1997) concluded the reason for enhancing number sense by mental 
computation is that it provides students with an “intuitive feel for numbers and their various 
uses and interpretations.” It is this ‘feel’ that enables automaticity in individual’s use of 
numbers in everyday situations (McIntosh et al., 1997). Whether this ‘intuitive feel’ can be 
enhanced by digital technologies is uncertain, but the evidence towards improved mental- 
maths performance when using these technologies is growing (Miller & Robertson, 2009; 
2011; Main & O’Rourke, 2011). 

The approaches presented thus far may strike a chord for many teachers in Australian 
schools and assist in reimagining what maths lessons could look like. And while it is hard to 
dispute that today’s students are spending alarming amounts of time on screens in their 
homes (Green, Brady, Olafsson, Hartley, & Lumby, 2011; Houghton, et al., 2015), it is also 
clear that there is more going on when students use digital games than mindless 
entertainment. Nonetheless, the criterion for the selection of these classroom technologies has 
typically been a source of conjecture (Prensky, 2010; Purcell, 2005). 


Evaluating Digital Game-based Learning 

Veteran American education reformist Cuban (1993) identified a number of concerns 
expressed by teachers regarding the selection of emerging technologies including; the ease of 
its use, flexibility, reliability, the availability of technical support, whether the outcomes for 
the student were worth the time required to learn the technology, and whether the technology 
compromised the teachers’ management of the learning environment both behaviourally and 
academically. Johnson (2006) summarised Cuban’s concerns in a single question for 
educators: “Will it [the technology] facilitate my efforts to create an orderly learning 
environment and motivate students?” and concluded that, “digital technologies that fail to 
meet these criteria have a low probability of adoption” (p. 18). 

In pursuing more detailed evaluations of digital technology, a recent study conducted 
in the Netherlands by All et al., (2015) reviewed the feedback from 33 stakeholders in 
organisations that either used or supported users of DGBL (including DGBL researchers, 
higher education institutions, secondary and primary schools, e-leaming companies, public 
utility companies and large private companies). They assert that there were three categories 
of desired outcomes when using this type of learning approach; enhanced learning, increased 
motivation, and acknowledged efficiency outcomes. Within All et al.’s., (2015, p. 32) 
research, stakeholders reported differing levels of importance for these outcomes depending 
on their context. For example, corporate entities were focussed more on cost efficiencies and 
schools on time efficiencies; as such DGBL efficacy is considered context-dependent. All et 
al., (2015) attempted to ‘conceptualise and operationalise’ a definition of DGBL 
effectiveness, and concluded if it achieved high ratings in any outcomes associated with 
learning, motivation and efficiency (without diminishing any of these outcomes) compared to 
other instructional methods it was seen as effective. 
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Aims of the Research 

This research is the first large-scale empirical research in Australia focussed on 
measuring the efficacy of HGCs in primary classrooms. The researchers sought to investigate 
the use of the HGCs Nintendo DS lites with the software Dr Kawashima’s Brain Training in 
a number of Western Australian schools after the success of their earlier pilot study (Main & 
O’Rourke, 2011). The impetus for this research was the results of similar studies in Scotland 
by Miller and Robertson (2009; 2011), the awareness that overall there remains a need for 
continued research on digital game usage in formal settings (Kebritchi et al., 2010), and that 
existing research is derived from numerous digital game platforms with very few focussed on 
HGCs. 

The overarching question that the researchers sought to answer when embarking on 
this research was whether the HGCs Nintendo DS lites with the software Dr Kawashima’s 
Brain Training were more effective in developing students’ speed and accuracy in mental- 
maths than traditional classroom mental-maths instruction. In the context of this research, 
efficacy was determined by improvements in students’ scores on a timed assessment of basic 
math functions. In addition, the researchers were interested in what the students and teachers 
thought about using the HGCs for mental-maths. Including whether students were engaged 
when using the HGCs. Drawing from Shernoff s (2013) definition, students were deemed 
engaged when they were observed to be displaying interest, enjoyment and concentration 
simultaneously. 

To summarise the researchers’ intentions, it aligns with Kirriemuir and McFarlane 
(2004) cautionary thoughts on DGBL enquiry in that it should be a rigorous enquiry that can 
be replicated in other classroom settings, employ digital games that are not ‘fads’, and be 
used in conjunction with sound classroom practice to maximise learning outcomes. And 
further, it acknowledges the concerns of Bennet et al., (2008, p. 14) that evidence presented 
in regard to DGBL should ensure that it “includes the perspectives of young people and their 
teachers and that [it] genuinely seeks to understand the situation”. 


Method 

Ten classes of students (six Year 4, four Year 4/5) from seven different schools were 
involved in quasi-experimental design research to explore the use of Nintendo DS lites with 
the Dr Kawashima’s Brain Training software to improve speed and accuracy in mental-maths 
functions. 


Procedure 

After ethics approval had been granted by the researchers’ university, the WA 
Department of Education, and Catholic Education WA, the school principals of the selected 
schools were approached and the parameters of the research discussed. Upon principal 
approval, the researchers visited the seven classroom teachers to explain each school’s 
involvement. 

Students in the experimental (‘intervention’) classrooms used Nintendo DS lites and 
Dr Kawashima’s Brain Training for 20 minutes at the beginning of each day over a 10-week 
school term to develop their mental-maths skills. Dr Kawashima’s Brain Training game is 
based on research on the use of mental agility tasks to slow cognitive deterioration 
(Kawashima et al., 2005); however, the researcher’s reason for selecting this program was the 
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prevalence of games involving the rapid recall of mathematical facts. Dr Kawashima’s Brain 
Training involves several maths-related computation games that focus on speed and 
accuracy. 

The classroom teachers using HGCs were given an overview of the recommended 
classroom practice, including timeframes, setting responsibilities for class members in regard 
to charging and packing away the HGCs, and encouraging students to set goals in terms of 
speed and accuracy of mental-maths recall. The teachers were also given a Nintendo DS life 
console with the Dr. Kawashima’s Brain Training program prior to commencement of the 
school term so that they could familiarise themselves with the functionality of the console 
and the game. 

When using the HGCs the students were directed to spend at least half their daily 
sessions on a game, in which 20 random single digit addition, subtraction and multiplication 
sums appear on the screen. Students enter their answers using a stylus on the HGC touch 
screen and are given a time and a description of the game speed on completion of each set. 
Figure 1, for example, shows a student obtaining ‘rocket speed’ for completing the sequence 
under 10 seconds. There is no set level of difficulty for the 20 random questions and progress 
is gauged by speed and accuracy. After they had completed the x 20 session, students were 
free to choose from other maths-oriented games; one appealing element of Dr Kawashima’s 
Brain Training is that the more sessions played, the more games become available. The 
researchers and research-assistant visited the classrooms on day one of the intervention to 
ensure all students had set-up their HGCs appropriately, and to determine whether the 
intervention protocols were being adhered to. 

The teachers of the comparison classes were also visited by the researchers prior to 
the beginning of the new term and encouraged to commit to the ‘effective practice’ strategies 
and other highly engaging classroom teaching methods for mental-math, such as those 
identified by Swan (2007). 

Both groups were told that their students’ mental-maths recall would be measured at 
the beginning and the end of the term and to encourage students to set speed and accuracy 
goals for their daily 20-minute classes over the term. In the interest of equity, the comparison 
classes were given the HGCs to use in the following term. 


Participants 

There were two hundred and seventy eight Year 4 and 5 students, (aged between nine 
and eleven years) in ten classrooms from diverse socio-economic backgrounds in the Perth 
metropolitan area, involved in the study. Complete data were obtained for 236 students, of 
whom 120 students were female and 116 male. The seven schools were selected on the basis 
of proximity to the researchers’ university as well as approaches made by teachers after a 
conference presentation on the use of the HGCs (Main & O’Rourke, 2009) and media 
publicity following the success of the initial pilot study (Hiatt, 2009). Four schools were 
public schools and three were from the Catholic education system. Five classroom teachers 
were female and three were male (two of the teachers doubled up with separate classes). The 
teachers represented a cross section of experience and classroom styles. None of the teachers 
had previous experience implementing DGBL in classrooms prior to this study. 
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Data Collection 

The data collection methods included observations, semi-structured interviews and 
standardised assessments. As with the pilot-study (Main & O’Rourke, 2011), we employed 
the Westwood One Minute Test of Basic Number Facts to measure numeracy skills. The One 
Minute Test of Basic Number Facts (Westwood, 1987) is a norm referenced assessment 
consisting of four 33-item tests, one for each of the basic maths functions (+, - , x and -r), 
with a test-retest reliability of .88 to .92 according to each sub-test (Westwood, 2003). 
Students had one minute in which to complete the 33 questions for each mathematical 
function and the score for the overall test is determined by adding together each of the 
subtests, with the total score for the test being 132. At the end of the term prior to the 
research intervention, all participating students were given the Westwood One Minute Test of 
Basic Number Facts and were re-tested in the final week of the term in which the research 
took place. 

While measuring student performance is critical in determining the efficacy of the 
HGCs utilised in this research, this is not the only measure of educational value when judging 
the use of digital technology (Johnson, 2006). Beavis, Muspratt and Thompson (2015) assert 
the importance of gaining the perspectives of the students when evaluating the use of DGBL 
in classrooms and, in addition to observing and interviewing the students, we also sought 
feedback from teachers and parents on their perceptions of the impact of the HGCs on student 
engagement and performance.-Semi-structured interviews were conducted with a sample of 
36 students (four were selected by each of the classroom teachers to represent a cross section 
of abilities in the class) and eight classroom teachers: prior to the commencement of the 
intervention, during the course of the term, and on completion of the intervention. Only three 
parents were available to be interviewed. Interview questions were focused on identifying the 
interviewees’ opinions on using the HGCs as learning tools, student enjoyment of HGCs, 
changes in attitude to mathematics, concerns and apprehensions about using this technology, 
opportunities imagined with the HGC, and changes in teaching roles as a result of 
implementing the HGCs. 

The research assistant conducted weekly visits to observe the mental-maths sessions. 
Classroom observations were also conducted in the comparison classes to identify the 
teachers’ approaches to mental-maths and student engagement, but these observations were 
limited due to time constraints and were only undertaken to ensure that the students in these 
classes were receiving comparable time and instruction in mental-maths. 
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Data Analysis 

Parametric and non-parametric statistics were used to test for differences between the 
comparison and intervention classes on the One Minute Test of Basic Number Facts. 
Observations and interviews were examined for common themes using the open coding 
method established by Strauss and Corbin (1998). 

Key findings from the qualitative data on student engagement and the role of the 
teacher have been published in earlier papers (O’Rourke, Main, & Ellis, 2012; 2013), but are 
reported here with the previously unreported student achievement data to provide a more 
complete set of data from which to assess the efficacy of the DGBL used in this study. 
Further, these data are examined against the criteria identified by All et al., (2015, p. 29), to 
evaluate the efficacy of DGBL, and used to frame the discussion on the overall efficacy of 
the intervention. 


Findings 

Student achievement data is presented first as the primary research question was 
whether the HGCs Nintendo DS lites with the software Dr Kawashima’s Brain Training were 
more effective in developing students’ speed and accuracy in mental-maths than traditional 
classroom mental-maths instruction. This is followed by a summary of the qualitative data, 
including reporting on already published data and examining additional data, to elucidate 
perceptions of the intervention and student engagement. 


Analytic Approach 

To test for differences between the comparison and intervention classes on the Basic 
Number Facts test we used both parametric and non-parametric statistics. Analysis of 
covariance (ANCOVA), with pre-test as the covariate, was our preferred approach, because 
the comparison/intervention groups were based on intact classes rather than random 
assignment of individual students and it could not be assumed the two groups would be 
equivalent on the pre-test. 

Our choice of ANCOVA is consistent with much of the literature on the analysis of 
pre-test/post-test designs (e.g., Dimitrov & Rumrill, 2003; Owen & Froman, 1998; 
Rheinheimer & Penfield, 2001; Wright, 2006). However, not uncommonly for classroom- 
based research, the data did not meet two of the underlying statistical assumptions of 
ANCOVA: conditional normality (or normality of residuals) and homoscedasticity (equality 
of variances). These breaches were possibly due to a ceiling effect whereby students who 
achieved a perfect or near perfect score on the Basic Number Facts pre-test were not able to 
demonstrate any further improvement at post-test. Transformation of the data did not redress 
the problem. Although the ANCOVA is relatively robust against violations of statistical 
assumptions (Barrett, 2011; Field, 2006; Olejnik & Algina, 1984); the breaches reported 
herein do cast some doubt on the validity of the results and, as such, we also report the results 
of non-parametric Mann-Whitney U tests based on the students’ gain scores (i.e., post-test 
score minus pre-test score). 
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Improvement in Number Fact Nnowledge 

A graphic representation of the average gains from pre-test to post-test made by each 
class on the Basic Number Facts test is provided in Table 2, and the means and standard 
deviations for the combined comparison and intervention groups in Table 1. The apparently 
superior average gain made by the intervention group (20.41, versus 3.78 for the comparison 
group) was confirmed by the results of the parametric and non-parametric tests. Independent t 
test analysis of the pre-test scores revealed no significant difference between the two groups 
before commencing the intervention (t( 234 ) = -1.263 ,p = .208). At post-test, however, the 
intervention group performed significantly better than the comparison group (F(i, 233 ) = 

79.686, p < .001) after controlling for the effect of pre-test performance. This pattern was 
also borne out by the non-parametric analyses on students’ gain scores (Median gains: comp. 
= 5; exp. = 19; U = 2771, p < .001) using the Mann-Whitney U test. 

Further analysis was conducted using two-way ANCOVA to check for the possible 
influence of gender on the results. However, there was no significant main effect of gender 
(F( i, 23 i) = 1.895, p =.170), nor significant interaction between gender and group (F 1 . 231 , = 
1.256,= .264). 


a Comparison group 0ne Minute Basic Number Facts Test: 

■ Intervention group Average Gains (pre to post) 

40 


35 4 33.26 



Classes 


Figure 2: Average gains of Individual Classes on the Basic Number Facts Test. 
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Pre-test 


Post-test 



Gain 



mean 

(SD) 

mean 

(SD) 

Adj. mean* 

mean 

(SD) 

Intervention (n=124) 

57.34 

(21.43) 

77.75 

(27.91) 

79.70 

20.41 

(15.91) 

Comparison (n=l 12) 

61.23 

(25.88) 

66.79 

(28.26) 

64.64 

3.78 

(21.29) 

Test statistic 

f(234)=-1.263 (1 > 

F ( i,233) = 79.686® 

U= 2771® 

Significance 

p = .208 

p <.001 

p < .001 

Effect size 


rip 2 = .255 

r = .519 


* Adjusted mean 

(r> Independent samples f-test, pre-test only 

(2) ANCOVA, post-test with pre-test as covariate 

(3) Non-parametric Mann-Whitney U test, gain scores 

Table 1: Mean Pre-test, Post-test and gain Scores on the Basic Number Facts Test. 

Table 2 contains a breakdown of the group means for each of the Basic Number Facts 
sub-tests: addition, subtraction, multiplication and division. Results of ANCOVA using the 
pre-test scores as covariate, again show the intervention group performed significantly better 
on each sub-test than the comparison group. Non-parametric Mann-Whitney U tests on the 
sub-tests using gain scores also showed significant differences between the comparison and 
intervention groups (see Table 2). Further Table 3 highlights the median gains in each 
discipline area and again identifies significant differences between the groups. 



Means 

Addition 

Pre 

Post 

Subtractioi 

Pre 

1 

Post 

Multiplical 

Pre 

tion 

Post 

Division 

Pre 

Post 

Intervention 1 

21.09 

26.15 

16.76 

22.33 

12.83 

18.79 

6.68 

10.51 

Comparison 2 

21.79 

23.24 

16.77 

18.22 

13.48 

14.94 

9.20 

10.39 

ANCOVA 

F (i, 233 ) = 42.704 

F (i, 232 ) = 45.918 

F (1,233) = 63.476 

F 0 , 233 ) = 13.643 

Significance 

p < .001 

p < .001 

p < .001 

p < .001 


1 n = 124 

2 n = 112 


Table 2: Basic Number Facts sub-test means 



Median Gains 





Addition 

Subtraction 

Multiplication 

Division 

Intervention 1 

5 

5 

5 

3 

Comparison 2 

1 

0 

1 

1 

Mann-Whitney U 

3870.5 

3632.0 

3149.5 

5268.0 

Significance 

p < .001 

p < .001 

p < .001 

p = .001 


1 n = 124 

2 n = 112 


Table 3: Basic Number Facts sub-test median gains. 
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As noted earlier, for equity reasons the classes from the comparison group received 
the HGC intervention in the following school term. Their progress was measured at the end 
of the 10-week intervention using the Basic Number Facts test. Since three sets of test scores 
(i.e., pre, postl, post2) were available for these classes, we conducted a repeated measures 
analysis to determine whether the greater progress previously seen for the intervention group 
would be replicated. Because the data violated the assumptions of normality and sphericity, 
and data transformation did not redress the problem, we used a non-parametric Friedman test 
rather than repeated measures ANOVA. 

Complete data for the repeated measures analysis were available for 84 of the 
students. The results of the Friedman test showed a significant difference in test scores over 
time, x 2 (2) =112.663, p < .001. Post hoc comparisons were conducted using the Wilcoxon 
signed-rank test, with Bonferonni correction applied (.05/2) so that the significance level was 
set at p < .025. The medians and interquartile ranges (IQRs) for the pre, postl and post2 tests 
were 57.5 (44.25 to 71.0), 64.5 (46.5 to 78.0) and 81.0 (62.5 to 101.75), respectively. The 
effects of maturation meant there were significant differences between both the pre and postl 
tests (no intervention), Z = -5.251,p < .001, r = 0.33, and postl and post2 tests (after using 
HGCs), Z = -7.625, p < .001, r = .48). However, the results support our initial finding that the 
HGCs had a greater effect on the students’ mental-maths performance (as measured by the 
Basic Number Facts test) than traditional classroom approaches. There was no follow-up in 
regard to maintenance of these gains from the experimental groups as this was not within the 
scope of the research. 


Teacher Practice 

Classroom observations and interviews with teachers were seen as important sources 
of data in determining the impact of the intervention. Observations were undertaken in both 
the intervention and comparison classrooms to determine the potential influence of teacher 
practice. All classroom teachers involved in the study (in both intervention and comparison 
classrooms) were observed taking time to review student awareness of mathematical 
concepts, and encouraging students to share strategies with their peers at set times during the 
week (as per researcher guidelines). Teachers in the comparison classrooms were observed 
using modelling and explanations, drill and practice strategies, and completion of mental- 
maths worksheets. Specific strategies observed by the research assistant included rhyme and 
repetition, ‘rocks, papers, scissors’ exercises (where students in small groups held up the 
number of fingers in a group to answers times-table questions), and mini mental-maths 
quizzes using buzzers. These observations suggest that the teachers in the comparison 
classrooms were selecting strategies intended to engage students in mental-maths activities. 
Observation of a variety of teacher practice in the intervention classrooms (described by 
O’Rourke, Main, & Ellis, 2013) as ‘the guiding teacher’, ‘the facilitating teacher’ and the 
‘interactive teacher’), illustrated that regardless of the different teaching approaches when 
using the HGCs, these classroom pedagogies appeared to have little influence on the positive 
student perceptions and improvements in mental-maths skills. It was evident in classroom 
observations that by maintaining set classroom structures the HGCs had the capacity to 
engage and motivate students, irrespective of the approach of the teacher (O’Rourke et al., 
2013). 

The interviews with teachers were conducted on three occasions during the term and 
provided information on their perceptions of how the HGCs influenced their practice as well 
as that of their students. The semi-structured interviews were analysed for themes using a 
line-by-line open coding approach (Strauss & Corbin, 1998). The teachers’ feedback was 
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overwhelmingly positive with 64.4% of their comments relating to improved student 
engagement and motivation, enhanced problem solving, choice making, goal-setting and 
collaboration, and becoming more organised and independent (see O’Rourke, Main, & Ellis, 
2013 for full exposition of teacher feedback). While teachers were supportive of the 
intervention approach, 5% of interview material suggested that students could become overly 
competitive when using the HGCs. While the possible benefits and risks of competition in 
the classroom are not within the scope of this paper, competition (or self-competition) is an 
element of typical game playing and the students who enjoyed competition were motivated 
by this aspect of the intervention. Some teachers (2% of key themes) also noted technical 
issues when using the HGCs, but these concerns were alleviated by researcher support and on 
occasion, student problem solving. 

Of the comments affirming the use of the HGCS, 24% were directed towards the 
changing roles of teachers in classrooms employing digital game technology. These roles 
have been explored in more detail in O’Rourke, Main, and Ellis (2013), but for one teacher 
the HGCs provided an assurance of quality instruction: 

[The DSs] Just makes learning their tables fun. Saves me doing it and 
whichever way teachers try to do it...it’s all pretty time consuming and it’s all 
pretty boring. This way [using the DSs] I can basically keep an eye on them and 
they go for it... 

As an indicator of teachers’ perceptions of the intervention, several of the teachers 
requested using the class-sets of HGCs in the following years indicating positive affirmation 
of this process of learning. 


Student Engagement 

In order to answer the research question pertaining to student engagement, interviews 
with students were recorded, transcribed then analysed using an open-coding method (Strauss 
& Corbin, 1998) common themes, patterns and tensions were revealed. The following 
vignette recorded by the research assistant, who visited classrooms on a weekly basis, 
provides a snapshot of the level of student engagement and classroom behaviour when 
students were using the HGCs: 

8.50 am: “Oh yeah we are doing DS”, chorus the students. General discussion 
on scores and personal bests. Student X says he is catching up to the classroom 
teacher. Distribution of DSs - mostly done individually, some picked them up for 
their friends. Very little teacher direction. Students are very well organised - 
folders are out and DSs ready to go. Students stand up when all set. Student X 
and Y prepare for battle. Instructions simple, no issues of behaviour. Four 
students undertaking Sudoku, square games while others are completing x 20. 

9.00 am: Students still concentrating on tasks. One student distracted, not 
feeling well. 

9.02 am: Postures of students range from normal upright position, to legs 
pressed against the table top, turned in chair with head on back of chair, sitting 
diagonally to slumped positions. Comments of the different strategies used for 
faster times: writing smaller, getting in the zone. 

9.05 am: Sharing of scores by a few when completing a round. Faces of 
excitement when achieving a good score. Faces of disbelief when achieving an 
inadequate score. 

9.07 am: Students complete paper work; time sheet (students have been filling 
these in themselves each week); what I need to remember (little recorded here); 
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and scrap paper to record times - they are crossed out when they achieve a 
better time. 

9.10 am: Time is up. Students finish games and pack away quietly and orderly. 
Further themes that emerged from the semi-structured interviews with students (see 
Figure 2) were that the HGCs: were fun, easy to use, novel, challenging, provided choice and 
allowed students to develop strategies (and subsequently self-regulate their efforts) (see 
O’Rourke, Main, & Ellis, 2012 for the full analysis of this data). Although limited, there was 
negative feedback from students related to technical issues. Specifically, the frustration they 
experienced when the Dr Kawashima’s Brain Training program did not recognise their 
writing and therefore indicated a wrong answer. Additionally, some students recognised that 
the format of the game resulted in their classmates becoming overly competitive. As 
previously stated, the issue of competition in the classroom is complicated and beyond the 
scope of this paper; however, it is acknowledged that the effects of competition can be 
anxiety inducing for many students (PISA, 2015) and as such needs to be given 
consideration. 


Motivated and encouraged practice 

Fun 

Improves speed 
Improves maths strategy | 

Novel 

Improves confidence 
Technical issues I 
Improves game strategy 
Provides variety 
Individualises 
Too much competition 

0 

Figure 2: Sub-themes from interviewed students derived from open-coding method (85 interviews 

reviewed). 

Some students appeared genuinely surprised by their improvement in mental-maths 
computations using the HGCs and this suggested movement towards the ‘intuitive feel’ 
identified by McIntosh et al., (1997). 

Student A: Well I’m learning well, I’ve wrote [sic] the sums quicker than I ever 
have before, like, when we ’re doing a sheet, I don ’t do it as quick for some 

reason, it’s just when you ’re so concentrated on . you just do them. 

Student B: I know I can do it and it’s just I have a feeling to get past it! 

Only three parents were available to be interviewed to determine their perceptions on 
their child’s use of the HGCs in the classroom. The limited number of parents available for 
interview means that their feedback may not be indicative of all parents’ views. Parents’ 
responses were interrogated to identify areas of concern and indications of student 
engagement with the HGCs. There were initial concerns about the potential lack of strategic 
input from teachers, but once the intervention commenced it became evident to parents that 
teachers were providing this input and this issue was not raised in subsequent interviews. 

One parent did express concern that Brain Training was directed toward more surface 
learning, when a deeper appreciation of mathematical concepts would have been beneficial, 
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but this is a criticism that could be levied at other forms of drill and practice used to teach 
basic math concepts. In subsequent interviews all parents reported that their children were 
motivated by the HGCs and that their attitude to maths had become more positive (O’Rourke, 
Main, & Ellis, 2012). 


Discussion 

The purpose of this study was to examine whether the use of a digital game using Dr 
Kawashima’s Brain Training could increase Year 4 student engagement and automaticity in 
mental-maths in seven different primary schools in Perth, Western Australia. To provide a 
framework for assessing the effectiveness of this approach, a brief summary of the three 
desired outcomes of All et al’s., (2015) conceptual framework for assessing the effectiveness 
of DGBL is provided. All et al., (2015, p. 29) proposed that operationally DGBL was 
effective when users “achieve similar or higher scores compared to other instructional 
methods” via: 

1. Enhanced learning outcomes as a result of increased interest in the subject matter, 
improvement in performance or transfer of skills to real-life situations. 

2. Increased motivational outcomes through enjoyable experiences of playing the game, 
or increased enthusiasm toward use of the DGBL, and 

3. Acknowledged efficiency outcomes through time-management and cost-effectiveness. 
Results of this study and previous research (Main & O’Rourke, 2011; O’Rourke, 

Main, & Ellis, 2012; 2013) indicate that use of the HGCs achieved most of All et al’s., (2015) 
desired outcomes. 

Given the significant statistical improvements in mental-maths skills of the students in 
the HGC intervention classrooms, when compared to those in the comparison classrooms, the 
qualitative feedback from students on enjoyment of using the HGCs, improvement in speed 
and accuracy, and motivation/enthusiasm towards learning, outcomes one and two of All et 
al’s., (2015) framework appear to be satisfied. Teachers too were satisfied with their students’ 
improvements in mental-maths accuracy and capacity to strategise. Further, there was clear 
evidence that the implementation of the HGCs in the intervention classes resulted in transfer 
of skills to real-life applications and settings via student feedback and improvements in 
mental-maths recall in standardised testing situations. 

While schools’ perspectives on ‘cost-effectiveness’ were not measured it is worth 
reflecting on that none of the schools purchased a class-set of the HGCs post the 
interventions. Several schools asked to use the class-sets again, but school administrators did 
not see purchasing HGCs as an investment appropriate for school funds. While this may 
indicate that the schools did not see the HGCs as cost effective in terms of meeting the needs 
of their student’s learning, De Grove, Bourgonjon and Van Looy (2013) point out that 
schools have limited budgets for technology and as such principals may be more inclined to 
purchase equipment that has perceived stronger utility in a number of curricular materials. 

Finally, it could be argued that the HGC intervention provided classrooms with time 
efficiencies (and thus met at least part of All’s et al., (2015) third outcome of effective 
DGBL) via enhanced student engagement. In an earlier pilot study by Main and O’Rourke 
(2011) exploring the implementation of the HGC, video footage (taken three times during the 
term) and classroom observations were analysed to determine classroom behaviour. An 
overall estimate of the time students spent on engaged/non-engaged behaviours was obtained 
by amalgamating and averaging the coding data. This data indicated that students were, on 
average, engaged (using the HGCs on their own) for 65% of the 20 minute session with only 
10 % of the time being spent in non-specific activities (off-task behaviour and conversations 


Vol 42, 10, October 2017 


63 



Australian Journal of Teacher Education 


not related to the use of the HGCs). Of the remaining time, approximately 15% was spent on 
sharing (student shares their progress with the teacher and their peers, often in the form of 
high scores) and 10% on assisting (student assists another with the operation of the HGC e.g., 
student takes a leadership role in collaborating with fellow students to address technical 
issues and strategies [see O’Rourke, Main, & Ellis (2013) for a full exploration of student 
classroom behaviour]). While such analysis was not conducted in the current research, the 
vignette presented in the results section challenged the perception that many students in 
primary settings meet maths classes with antipathy and non-enthusiasm. In this case the 
classrooms observed (along with others using the HGCs), had well-structured and time 
focussed lesson protocols, universal design for learning features (via student choice making), 
strategy development, student collaborations and lesson reflections. These antecedents impact 
on the “amount of time and energy a student is willing to devote to learning” (Churchill et al., 
2011, p. 124) and in combination with a popular digital tool such as the Nintendo DS lite can 
result in academic gains. Overall, the students’ use of the HGCs resulted in a high degree of 
student engagement and significant improvement in their addition, subtraction, multiplication 
and division over the school term. 


Limitations 

While the results of this research support the implementation of HGCs in primary 
classrooms, we are aware there are many variables that could have impacted on the students’ 
mathematical improvement and these limit the generalisations that can be made from this 
study. In this research the mental-maths approaches undertaken in the comparison classes 
could not be controlled, which ‘muddies the water’ for the comparison statistics presented 
(All, et al., 2014). Nonetheless, because we closely followed the protocols of an earlier pilot 
study (Main & O’Rourke, 2011) and similar research conducted in Scotland, the consistent 
results are promising. Further, while the authors do not see this as a ‘cure-all’ for disengaged 
primary aged maths students, they recognise this approach aligns with contemporary 
Universal Design for Learning (UDL) approaches (Rose & Meyer, 2002) and is an 
intervention that, if used strategically, is appropriate in a variety of settings (Main, O’Rourke, 
Morris, & Dunjey, 2016). 

As O’Neil, et al., (2005, p. 465) described: “Games themselves are not sufficient for 
learning, but there are elements in games that can be activated within an instructional context 
that may enhance the learning process.” Thus, it is the digital game in combination with 
classroom exigencies that impact on student learning. Classroom observations provided 
additional information on the situation under which the HGCs were used, which indicated 
that implementation and outcomes were similar even when the teaching style differed. 
Regardless of these differences (O’Rourke, Main, & Ellis, 2013), the teacher’s role was still 
vital in supporting students with strategies to be successful with their mental math (e.g., 
unpacking mental-maths strategies on the board, acknowledging and encouraging student 
improvement, assisting students to self-regulate their mental-maths progress). 

All et al., (2015, p. 37) suggest, “a DGBL intervention is effective when it achieves 
similar or higher scores in comparison to ‘business as usual’ in relation to [sic] learning, 
motivational or efficiency outcomes, without significantly diminishing any of the others.” It 
is acknowledged that further testing to determine if gains were maintained by the intervention 
group would have solidified the findings, but the collected research data on improved mental- 
maths performance over the term, along with positive feedback from students and teaching 
staff on enhanced learning, motivation and classroom efficiencies, indicate that the HGCs 
align (in the most part) with All et al’s conceptual framework on DGBL efficacy, and provide 
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the impetus for future investment in interventions using commercially available digital games 
in the maths classroom. 


Conclusion 

Despite the limitations of this research, we contend the findings demonstrate that 
commercially available digital games, such as the Nintendo DS life, can be useful tools in the 
primary classroom. The Nintendo DS lite was simply one of the technologies engaging 
children at the time of the research, but thoughtful implementation of this technology resulted 
in academic gains for the students using it. Cuban’s (1993) concerns about technology in the 
classroom, namely, whether it was easy to use, whether support was needed and available, 
whether the outcomes for the student were worth the time required to learn the technology, 
and if the technology had an impact on the teachers’ ability to manage the learning 
environment both behaviourally and academically; all appeared to be addressed within this 
classroom intervention. Perhaps it is all summarised succinctly, by one of the interviewed 
students who, when asked why he preferred the HGC over non-digital approaches to 
mathematics, simply responded: “Because it’s more fun - it encourages you to do more!” 
Further, as Elton-Chalcraft and Mills (2013) in their exploration of creative and fun 
curriculum, found: “Children, students and teachers all agreed learning is more effective 
when children have more control over their learning and when activities and learning 
environment were both fun and challenging” (p. 495). Thus, the implication for teachers, and 
teacher educators, is to look beyond technology specifically designed for classrooms to the 
technology that our students choose to access away from these settings. In this space we may 
find tools that are easy for our students to access independently, provide fun and challenge, 
and result in desired learning outcomes. 
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